Skip to content

Feature: EventPreprocessor#2928

Merged
maxnoe merged 19 commits into
mainfrom
feature/event_preprocessor
Mar 3, 2026
Merged

Feature: EventPreprocessor#2928
maxnoe merged 19 commits into
mainfrom
feature/event_preprocessor

Conversation

@kosack
Copy link
Copy Markdown
Member

@kosack kosack commented Jan 28, 2026 •
edited
Loading

Introduces an EventPreprocessor class that can generically transform an event table by applying the following steps:

  • Generate new or rename existing columns with a FeatureGenerator
  • Select "good" event rows with a QualityQuery
  • Select which columns to output (by setting the features configuration attribute of the EventPreprocessor)

This is useful for doing the final steps of DL2 processing, and will eventually replace what is in DL2EventPreprocessor and DL2EventLoader, which will be deprecated in a future release.

The EventPreprocessor also includes the ability to pre-configure itself for specific use cases by setting the feature_set option. Currently only two ProcessingFeatureSet are implemented: feature_set=dl2_irf, which defines the transforms, event selection, and output features for processing simulated DL2 events, and feature_set=custom, which has no pre-configuration and requires all parmeters to be set in a config file.

The functionality of DL2EventLoader can be mimicked with the following:

from ctapipe.io import TableLoader, EventPreprocessor
from astropy.table import vstack

DL2FILE = "some_dl2_file.h5"
with  TableLoader(DL2FILE, dl2=True, simulated=True, observation_info=True) as loader:
    preprocess = EventPreprocessor(feature_set="dl2_irf") # or pass in a full config
    events = vstack(
        [
            preprocess(c.data) 
            for c in loader.read_subarray_events_chunked(chunk_size=100_000)
        ]
    )

tab = preprocess.quality_query.to_table(functions=True)
tab.reverse()
plt.barh(
    width=tab["cumulative_counts"],
    y=tab["criteria"],
)
plt.xlabel("Cumulative Counts")
image

this comes from the refactoring of DL2EventLoader in #2919

An example configuration, equivalent to the dl2_irf feature_set is:

 EventPreprocessor:
  FeatureGenerator:
    features:  # to generate
    - - reco_energy
      - RandomForestRegressor_energy
    - - reco_alt
      - HillasReconstructor_alt
    - - reco_az
      - HillasReconstructor_az
    - - gh_score
      - RandomForestClassifier_prediction
    - - theta
      - angular_separation(reco_az, reco_alt, true_az, true_alt)
    - - reco_fov_coord
      - altaz_to_fov(reco_az, reco_alt, subarray_pointing_lon, subarray_pointing_lat)
    - - reco_fov_lon
      - reco_fov_coord[:,0]
    - - reco_fov_lat
      - reco_fov_coord[:,1]
    - - true_fov_coord
      - altaz_to_fov(true_az, true_alt, subarray_pointing_lon, subarray_pointing_lat)
    - - true_fov_lon
      - true_fov_coord[:,0]
    - - true_fov_lat
      - true_fov_coord[:,1]
    - - true_fov_offset
      - angular_separation(reco_fov_lon, reco_fov_lat, 0*u.deg, 0*u.deg)
    - - reco_fov_offset
      - angular_separation(true_fov_lon, reco_fov_lat, 0*u.deg, 0*u.deg)
    - - multiplicity
      - np.count_nonzero(RandomForestClassifier_telescopes,axis=1)
  QualityQuery:
    quality_criteria:  # can use any input or generated feature
    - - Valid geometry
      - HillasReconstructor_is_valid
    - - valid energy
      - RandomForestRegressor_is_valid
    - - valid gammaness
      - RandomForestClassifier_is_valid
    - - sufficient multiplicity
      - multiplicity >= 4
  features: # to output after selection
  - event_id
  - obs_id
  - reco_energy
  - reco_alt
  - reco_az
  - gh_score
  - true_energy
  - true_alt
  - true_az
  - true_fov_offset
  - reco_fov_offset
  - theta
  - reco_fov_lat
  - true_fov_lat
  - reco_fov_lon
  - true_fov_lon
  - multiplicity

maxnoe
maxnoe reviewed Jan 28, 2026
View reviewed changes
Comment thread docs/changes/2928.feature.rst Outdated
maxnoe
maxnoe reviewed Jan 28, 2026
View reviewed changes
Comment thread src/ctapipe/coordinates/tests/test_utils.py Outdated
@kosack kosack force-pushed the feature/event_preprocessor branch from 11b0493 to 87b43dc Compare January 28, 2026 13:44
maxnoe
maxnoe reviewed Jan 28, 2026
View reviewed changes
Comment thread src/ctapipe/coordinates/utils.py Outdated
maxnoe
maxnoe reviewed Jan 28, 2026
View reviewed changes
Comment thread src/ctapipe/io/event_preprocessor.py Outdated
maxnoe
maxnoe reviewed Jan 28, 2026
View reviewed changes
Comment thread src/ctapipe/io/event_preprocessor.py Outdated
maxnoe
maxnoe reviewed Jan 28, 2026
View reviewed changes
Comment thread src/ctapipe/io/event_preprocessor.py Outdated
@kosack kosack force-pushed the feature/event_preprocessor branch from 720e800 to 096fd64 Compare January 28, 2026 15:05
@kosack kosack force-pushed the feature/event_preprocessor branch from add280f to 88657f5 Compare January 28, 2026 16:22
maxnoe
maxnoe previously approved these changes Feb 2, 2026
View reviewed changes
@kosack kosack requested a review from mexanick February 3, 2026 10:56
@maxnoe maxnoe requested a review from mdebony February 11, 2026 13:44
@maxnoe
Copy link
Copy Markdown
Member

maxnoe commented Feb 11, 2026

@mdebony This is intended for DL2 to DL3, so please take a look here.

I think it would be good to get this in and then use it in the DL2 to DL3 tool

@mexanick mexanick requested a review from Voutsi February 12, 2026 14:48
Voutsi
Voutsi reviewed Feb 12, 2026
View reviewed changes
Comment thread src/ctapipe/io/event_preprocessor.py Outdated
Voutsi
Voutsi reviewed Feb 12, 2026
View reviewed changes
Comment thread src/ctapipe/io/event_preprocessor.py Outdated
mexanick
mexanick reviewed Feb 12, 2026
View reviewed changes
Comment thread docs/changes/2928.feature.rst Outdated
Comment thread src/ctapipe/io/event_preprocessor.py Outdated
@kosack kosack requested review from Voutsi and mexanick February 19, 2026 15:12
mexanick
mexanick approved these changes Feb 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@mexanick mexanick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ctao-sonarqube
Copy link
Copy Markdown

ctao-sonarqube Bot commented Feb 19, 2026

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

mdebony
mdebony requested changes Feb 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@mdebony mdebony left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems very usefull for DL3 production, I've added a small suggestion in the comments.

Comment thread src/ctapipe/io/event_preprocessor.py
This was referenced Feb 23, 2026
Open
Open
@mexanick
Copy link
Copy Markdown
Contributor

mexanick commented Mar 3, 2026

@mdebony could this be moved on?

@mdebony
Copy link
Copy Markdown
Contributor

mdebony commented Mar 3, 2026

Yes, it's can move on.

@mexanick mexanick requested a review from maxnoe March 3, 2026 13:33
@maxnoe maxnoe merged commit 13f0412 into main Mar 3, 2026
13 checks passed
@maxnoe maxnoe deleted the feature/event_preprocessor branch March 3, 2026 14:18
@LukasBeiske LukasBeiske mentioned this pull request Apr 2, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxnoe maxnoe maxnoe approved these changes

@mexanick mexanick mexanick approved these changes

@Voutsi Voutsi Voutsi approved these changes

@mdebony mdebony mdebony approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kosack @maxnoe @mexanick @mdebony @Voutsi